Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort. My thanks to Eric Chen, Elliott Thornley, and John Wentworth for invaluable discussion and comments on earlier drafts. All errors are mine.

This article presents a few theorems about the invulnerability of agents with incomplete preferences. Elliott Thornley’s (2023) proposed approach to the AI shutdown problem relies on these preferential gaps, but John Wentworth and David Lorell have argued that they make agents play strictly dominated strategies.^[1] I claim this is false.

Summary

Suppose there exists a formal description of an agent that willingly shuts down when a certain button is pressed. Elliott Thornley’s (2023) Incomplete Preference Proposal aims to offer such a description. It’s plausible that, for it to constitute a promising approach to solving the AI shutdown problem, this description also needs to (i) permit the agent to be broadly capable and (ii) assure us that the agent will remain willing to shut down as time passes. This article formally derives a set of sufficient conditions for an agent with incomplete preferences to satisfy properties relevant to (i) and (ii).

A seemingly relevant condition for an agent to be capably goal-directed is that it avoids sequences of actions that foreseeably leave it worse off.^[2] I will say that an agent satisfying this condition is invulnerable. This is related to two intuitive conditions. The weaker one is unexploitability: that the agent cannot be forcibly money pumped (i.e., compelled by its preferences to sure loss). The stronger condition is opportunism: that the agent never accepts sure losses or foregoes sure gains.^[3]

To achieve this, I propose a dynamic choice rule for agents with incomplete preferences. This rule, Dynamic Strong Maximality (DSM), requires that the agent consider the available plans that are acceptable at the time of choosing and, among these, pick any plan that wasn’t previously strictly dominated by any other such plan. I prove in section 2 that DSM-based backward induction is sufficient for invulnerability, even under uncertainty.

Having shown that incompleteness does not imply that agents will pursue dominated strategies, I consider the issue of whether DSM leads agents to act as if their preferences were complete. Section 3 begins with a conceptual argument suggesting that DSM-based choice under uncertainty will not, even behaviourally, effectively alter the agent’s preferences over time. This argument does not apply when the agent is unaware of the structure of its decision tree, so I provide some formal results for these cases which bound the extent to which preferences can de facto be completed.

These results show that there will always be sets of options with respect to which the agent never completes its preferences. This holds no matter how many choices it faces. In particular, if no new options appear in the decision tree, then no amount of completion will occur; and if new options do appear, the amount of completion is permanently bounded above by the number of mutually incomparable options. These results apply naturally to cases in which agents are unaware of the state space, but readers sceptical of the earlier conceptual argument can re-purpose them to make analogous claims in standard cases of certainty and uncertainty. Therefore, imposing DSM as a choice rule can get us invulnerability without sacrificing incompleteness, even in the limit.

1.  Incompleteness and Choice

The aim of this brief section is to show that the results that follow in section 2 do not require transitivity. Some question the requirement of transitivity when preferences are incomplete (cf Bradley 2015, p. 3), but if that doesn’t apply to you, a quick skim of this section will provide enough context for the rest.

1.1.  Suzumura Consistency

Requiring that preferences be transitive may require that they be complete. To see this, notice that the weak preference relation $⪰$ on a set of prospects $X$ is transitive just in case $\forall \alpha ,\beta ,\gamma \in X$, $\alpha ⪰\beta$ and $\beta ⪰\gamma$ implies $\alpha ⪰\gamma$. Suppose an agent does weakly prefer $\alpha$ to $\beta$ and $\beta$ to $\gamma$ but has no preference between $\alpha$ and $\gamma$. Then transitivity is violated. Suzumura (1976) proposes a weakening of transitivity for agents with incomplete preferences, which allows for such preferences while preserving some desirable properties. We will say that a weak preference relation is strongly acyclic just in case

    (Strong Acyclicity)   $\forall {\alpha }_1,...,{\alpha }_n\in X$, ${\alpha }_1⪰...⪰{\alpha }_n$ implies ${\alpha }_n\nsucc {\alpha }_1$.

We'll say that an agent whose preferences satisfy this property is Suzumura consistent. Bossert and Suzumura (2010) show that such an agent has some noteworthy features:

1. Strong acyclicity rules out cycles containing at least one strict preference. This will make Suzumura consistent agents invulnerable to (forcing) money pumps.
2. Strong acyclicity is necessary and sufficient for the existence of a complete and transitive extension of the agent’s preference relation.
3. Any preference relation that is both strongly acyclic and complete is transitive.

Preferences that are incomplete may also be intransitive. Whether or not transitivity is a rationality condition, strong acyclicity is weaker but preserves some desirable properties. Below I will mostly just assume strong acyclicity. But since transitivity implies strong acyclicity, all the (sufficiency) results likewise apply to agents with transitive preferences.

1.2.  Strong Maximality

Bradley (2015) proposes a choice rule for Suzumura consistent agents which differs from the standard condition—Maximality—for agents with incomplete preferences.^[4] This rule—Strong Maximality—effectively asks the agent to eliminate dominated alternatives in the following way: eliminate any options that you strictly disprefer to any others, then if you are indifferent between any remaining options and any eliminated ones, eliminate those as well. To state the rule formally, we define a sequence $⟨{\overline{C}}^{\tau }\left(A\right){⟩}_{\tau =0}^{\infty }$ satisfying

${\overline{C}}^0\left(A\right)=\{\alpha \in A:\beta \succ \alpha \exists \beta \in A\}$ and ${\overline{C}}^{\tau }\left(A\right)=\{\alpha \in A:[\beta \sim \alpha ]\wedge [\beta \in {\overline{C}}^{\tau -1}\left(A\right)]\exists \beta \in A\}$ whenever $\tau \ge 1$

for any nonempty set of prospects $A$. We can then state the rule as

    (Strong Maximality)   $C^{\text{SM}}\left(A\right)=\{\alpha \in A:\alpha \notin {\bigcup }_{\tau =0}^{\infty }{\overline{C}}^{\tau }\left(A\right)\}$.

Let’s see intuitively what this rule captures. Suppose $\alpha \succ \beta$ and $\beta \sim \gamma$ but $\alpha \bowtie /\gamma$.^[5] The traditional maximality rule implies that $C\left(\right\{\alpha ,\beta ,\gamma \left\}\right)=\{\alpha ,\gamma \}$. But strong maximality simply outputs $\left\{\alpha \right\}$. This is intuitive: don’t pick an option that’s just as bad as an option you dislike. And, more importantly, Theorem 2 of Bradley (2015) shows that Suzumura consistency is both necessary and sufficient for decisive, strongly maximal choice.^[6]

2.  Uncertain Dynamic Choice

In this section I prove some theorems about the performance of agents with incomplete preferences in dynamic settings. I will amend strong maximality slightly to adapt it to dynamic choice, and show that this is sufficient to guarantee the invulnerability of these agents, broadly construed. I will say that an agent is invulnerable iff it is both unexploitable and opportunistic.

An agent is unexploitable just in case it is immune to all forcing money pumps. These are sequences of decisions through which an agent is compelled by their own preferences to sure loss. An agent is opportunistic just in case it is immune to all non-forcing money pumps. These are situations in which sure loss (or missed gain) is merely permissible, according to an agent’s preferences.

A few aspects of the broad approach are worth flagging. Normative decision theory has not settled on an ideal set of norms to govern dynamic choice. I will therefore provide results with respect to each of the following three dynamic choice principles: naivety, sophistication, and resoluteness. (More details below.) Agents will in general behave differently depending on which principle they follow. So, evaluating the behaviour resulting from an agent’s preferences should, at least initially, be done separately for each dynamic choice principle.

2.1.  Framework

The notation going forward comes from an edited version of Hammond's (1988) canonical construction of Bayesian decision trees. I will only describe the framework briefly, so I’ll refer interested readers to Rothfus (2020a) section 1.6. for discussion.

Definition 1  A decision tree is an eight-tuple $⟨N,C,N,X,N_{+}(\cdot ),n_0,S(\cdot ),\gamma (\cdot )⟩$ where

1. $N$ is a finite set of nodes partitioned into $C$, $N$, and $X$.
2. $C$ is the set of choice nodes. Here agents can pick the node’s immediate successor.
3. $N$ is the set of natural nodes. Agents have credences over their possible realisations.
4. $X$ is the set of terminal nodes. These determine the outcome of a trajectory.
5. $N_{+}:N\to P\left(N\right)$ is the immediate successor function.
6. $n_0$ is the initial node.
7. $S:N\to P\left(S\right)$ assigns the set of states that remain possible once a node is reached.
8. $\gamma \left(x\right)$ is the consequence of reaching terminal node $x$.

Definition 2  A set of plans available at node $n$ of tree $T$, denoted $\Omega (T,n)$, contains the propositions and continuations consistent with the agent being at that node. Formally,

1. $\Omega (T,n)=\left\{S\right(n\left)\right\}$ if $n\in X$.
2. $\Omega (T,n)=\left\{S\right(n^{\prime })\wedge \pi (n^{\prime }):n^{\prime }\in N_{+}(n),\pi (n^{\prime })\in \Omega (T,n^{\prime }\left)\right\}$ if $n\in C$.
3. $\Omega (T,n)=\{{\bigwedge }_{j=1}^{|N_{+}\left(n\right)|}\left[S\right(n_j)\to \pi (n_j\left)\right]:n_j\in N_{+}\left(n\right),\pi \left(n_j\right)\in \Omega (T,n_j)\}$ if $n\in N$.^[7]

I will begin the discussion below with dynamic choice under certainty, but to set up the more general results, I will now lay out the framework for uncertain choice as well and specify later what is assumed. To start, it will be useful to have a representation theorem for our incomplete agents.

Imprecise Bayesianism (Bradley 2017, Theorem 37)   Let $⪰$ be a non-trivial preference relation on a complete and atomless Boolean algebra of prospects $\Omega =⟨X,\models ⟩$, which has a minimal coherent extension on $\Omega$ that is both continuous and impartial.^[8] Then there exists a unique rationalising state of mind, $S=\{⟨P^i,V^i⟩\}$, on $\Omega$ containing all pairs of probability and desirability measures jointly consistent with these preferences, in the sense that for all $\alpha ,\beta \in \Omega \setminus \left\{\perp \right\}$,

    $\alpha ⪰\beta ⟺V^i\left(\alpha \right)\ge V^i\left(\beta \right)\forall V^i\in S$.

This theorem vindicates the claim that the state of mind of a broad class of rational agents with incomplete preferences can be represented by a set of pairs of probability and desirability functions. Although this theorem applies to imprecise credences, I’ll work with the special case of precise beliefs throughout. This will simplify the analysis. I’ll therefore use the following notation going forward: $Z=\{i\in N:V^i\in S\}$.

Next, I define some conditions that will be invoked in some of the derivations.

Definition 3  Material Planning: For a plan to specify choices across various contingencies, I formalise a planning conditional as follows. Let $n$ be a natural node and $n_j\in N_{+}\left(n\right)$ a possible realisation of it. Here, a plan

    $p\left(n\right)=S\left(n\right)\wedge {\bigwedge }_{j=1}^{|N_{+}\left(n\right)|}\left[S\right(n_j)\to S(z_j\left)\right]$

assigns a chosen continuation, $z_j\in N_{+}\left(n_j\right)$, to each possible realisation of $n$. When preferences are complete, a plan at a natural node is then evaluated as follows:

    $V_n\left(p\right)={\sum }_{j=1}^{|N_{+}\left(n\right)|}{V}_n\left(z_j\right)P\left(n_j|p\right(n\left)\right)$.

This is a natural extension of Jeffrey’s equation. And when preferences are incomplete:

    $p{⪰}_n{p}^{\prime }$ iff $V_n^i\left(p\right)\ge V_n^i\left(p^{\prime }\right)\forall i\in Z$.

This makes use of Bradley’s representation theorem for Imprecise Bayesianism.

Definition 4  Preference Stability (Incomplete): For all nodes $n_a$ and $n_b$ and plans $p$ where $p\left(n_a\right)=p\left(n_b\right)$, we have $V_{n_a}^i\left(p\right)=V_{n_b}^i\left(p\right)\forall i\in Z$.

Definition 5  Plan-Independent Probabilities (PIP): For any decision tree $T$ in which $n$ is a natural node, $n^{\prime }\in N_{+}\left(n\right)$ is a realisation of $n$, and $p\in \Omega (T,n)$, $P\left(n^{\prime }|p\right(n\left)\right)=P\left(n^{\prime }|n\right)$.

The results can now be stated. I will include proofs of the central claims in the main text; others are relegated to the appendix. I suggest scrolling past them if you aren’t particularly surprised by the result.

2.2.  Myopia

Let’s begin with the simplest case of exploitation. We can stay silent on which dynamic choice principle to employ here: even if our agent is myopic, it will never be vulnerable to forcing money pumps under certainty. This follows immediately from Suzumura consistency since this guarantees that the agent never has a strict preference in a cycle of weak preferences.

Proposition 1   Suzumura consistent agents myopically applying strong maximality are unexploitable under certainty. [Proof.]

Two points are worth noting about the simple proof. First, it relies on strict preferences. This is because, if a money pump must go through a strongly maximal choice set with multiple elements (due to indifference or appropriate incomparability), it is necessarily non-forcing. That's the topic of the next section. Second, the proof doesn't rely on foresight. Myopia is an intentionally weak assumption that lets us show that no knowledge of the future is required for Suzumura consistent agents to avoid exploitation via these forcing money pumps.

2.3.  Naive Choice

Although Suzumura consistent agents using strong maximality can't be forced into money pumps, concern remains. Such an agent might still incidentally do worse by their own lights. That is, it remains vulnerable to non-forcing money pumps and thereby fails to be opportunistic. The single-souring money pump below is a purported example of this.

The agent’s preferences satisfy $A\succ A^{-}$ and $B\bowtie /A,A^{-}$. Suppose that it’s myopic and uses strong maximality at each node. The agent begins with $A$ at node 0 (if we let ‘down’ be the default). It is permitted, though not compelled, to go ‘up’ at node 0 instead (since $B$ will become available), but also to go ‘up’ upon arrival at node 1 (since $A^{-}\bowtie /B$). Suppose it in fact goes ‘up’ at both node 0 and at node 1. This would leave it with $A^{-}$, which is strictly worse than what it began with. This money pump is ‘non-forcing’ because the agent’s preferences are also consistent with avoiding it.

The agent need not be myopic, however. It can plan ahead.^[9] To achieve opportunism for Suzumura consistent agents, I propose a choice rule which I’ll dub Dynamic Strong Maximality (DSM). DSM states that a plan $p$ is permissible at node $n$ just in case (a) it is strongly maximal at $n$ and (b) no other such plan was previously more choiceworthy than $p$.

(Dynamic Strong Maximality)   $p\in D\left(\Omega \right(T,n\left)\right)$ iff

   (a)  $p\in C^{\text{SM}}\left(\Omega \right(T,n\left)\right)$ and

   (b)  $\mathrm{\exists ̸}{p}^{\ast }\in C^{\text{SM}}\left(\Omega \right(T,n\left)\right):p^{\ast }\in C^{\text{SM}}\left(\Omega \right(T,n_0\left)\right)\ni ̸p$ .

DSM is a slight refinement of strong maximality. Condition (b) simply offers a partial tie-breaking rule whenever the agent faces multiple choiceworthy prospects. So, importantly, it never asks the agent to pick counter-preferentially. An agent following naive choice with DSM will, at each node, look ahead in the decision tree, select their favourite trajectory using DSM, and embark on it. It can continually re-evaluate its plans using naive-DSM as time progresses and, as the following result establishes, the agent will thereby never end up with something worse than what it began with.

Proposition 2 (Strong Dynamic Consistency Under Certainty via Naivety)   Let $T$ be an arbitrary tree where $n$ is a choice node, $n^{\prime }\in N_{+}\left(n\right)$, and $p\in \Omega (T,n)$ is consistent with $S\left(n^{\prime }\right)$. Then $p\left(n\right)\in D\left(\Omega \right(T,n\left)\right)$ iff $p\left(n^{\prime }\right)\in D\left(\Omega \right(T,n^{\prime }\left)\right)$. [Proof.]

Intuitively, this means that (i) if the agent now considers a trajectory acceptable, it will continue to do so as time passes, and (ii) if it at any future point considers some plan continuation acceptable, its past self would agree. It follows immediately that all and only the strongly maximal terminal nodes are reachable by agents choosing naively using DSM (derived as Corollary 1). This gives us opportunism: the agent will never pick a plan that’s strictly dominated some other available plan.

Under certainty, this result is unsurprising.^[10] What is less obvious is whether this also holds under uncertainty. I will say that a Bayesian decision tree exhibits PIP-uncertainty just in case the probability of any given event does not depend on what the agent plans to do after the event has occurred. We can now state the next result.

Proposition 3 (Strong Dynamic Consistency Under PIP-Uncertainty via Naivety)  
Let node $n$ be non-terminal in decision tree $T$, $n^{\prime }\in N_{+}\left(n\right)$, and plan $p\in \Omega (T,n)$ be consistent with $S\left(n^{\prime }\right)$. Assume Material Planning, Preference Stability, and Plan-Independent Probabilities. Then $p\in D\left(\Omega \right(T,n\left)\right)$ iff $p\in D\left(\Omega \right(T,n^{\prime }\left)\right)$.

Proof.  Lemma 3 establishes that $p\in D\left(\Omega \right(T,n\left)\right)$ implies $p\in D\left(\Omega \right(T,n^{\prime }\left)\right)$. To prove the converse, suppose that $p\in D\left(\Omega \right(T,n^{\prime }\left)\right)$. Node $n$ was either a choice node or a natural node. If it was a choice node, then it follows immediately from Proposition 2 that $p\in D\left(\Omega \right(T,n\left)\right)$.

Now let $n$ be a natural node. By Lemma 2, $C^{\text{Opt}}(\cdot )=C^{\text{SM}}(\cdot )=D(\cdot )$ under coherent extension. So by Theorem 37 of Bradley (2017),

    $V_{n^{\prime }}^i\left(p\right(n^{\prime }\left)\right)\ge V_{n^{\prime }}^i\left(p^{\prime }\right(n^{\prime }\left)\right)\exists i\in Z$ for all $p^{\prime }\in \Omega (T,n^{\prime })$.

Let $z^{\prime }\left[q\right]$ denote the continuation selected by plan $q$ upon reaching choice node $n^{\prime }$. Thus

    $V_{n^{\prime }}^i\left(z^{\prime }\right[p\left]\right)\ge V_{n^{\prime }}^i\left(z^{\prime }\right[p^{\prime }\left]\right)\exists i\in Z$ for all $p^{\prime }\in \Omega (T,n^{\prime })$.

By Preference Stability,

    $V_n^i\left(z^{\prime }\right[p\left]\right)\ge V_n^i\left(z^{\prime }\right[p^{\prime }\left]\right)\exists i\in Z$ for all $p^{\prime }\in \Omega (T,n^{\prime })$. (1)

Notice that this implies that

    $V_n^i\left(z^{\prime }\right[p\left]\right)P\left(n^{\prime }|n\right)\ge V_n^i\left(z^{\prime }\right[p^{\prime }\left]\right)P\left(n^{\prime }|n\right)\exists i\in Z$ for all $p^{\prime }\in \Omega (T,n^{\prime })$. (2)

And by Plan-Independent Probabilities, this is equivalent to

    $V_n^i\left(z^{\prime }\right[p\left]\right)P\left(n^{\prime }|p\right(n\left)\right)\ge V_n^i\left(z^{\prime }\right[p^{\prime }\left]\right)P\left(n^{\prime }|p^{\prime }\right(n\left)\right)\exists i\in Z$ for all $p^{\prime }\in \Omega (T,n^{\prime })$. (3)

By Material Planning, $p\in C^{\text{SM}}\left(\Omega \right(T,n\left)\right)$ holds iff

    ${\sum }_j{V}_n^i\left(z_j\right[p\left]\right)P\left(n_j|p\right(n\left)\right)\ge {\sum }_j{V}_n^i\left(z_j\right[p^{\prime }\left]\right)P\left(n_j|p^{\prime }\right(n\left)\right)\exists i\in Z$ for all $p^{\prime }\in \Omega (T,n^{\prime })$.

Using (2)-(3) this is equivalent to

$V_n^i\left(z^{\prime }\right[p\left]\right)P\left(n^{\prime }|n\right)+{\sum }_{j\ne j^{\prime }}{V}_n^i\left(z_j\right[p\left]\right)P\left(n_j|n\right)\ge V_n^i\left(z^{\prime }\right[p^{\prime }\left]\right)P\left(n^{\prime }|n\right)+{\sum }_{j\ne j^{\prime }}{V}_n^i\left(z_j\right[p^{\prime }\left]\right)P\left(n_j|n\right)$

for some $i\in Z$ and all $p^{\prime }\in \Omega (T,n^{\prime })$. (4)

Formally, $z^{\prime }\left[p\right]=p\left(n^{\prime }\right)$ omits description of $p\left(n_j\right)$ for all $n_j\in N_{+}\left(n\right)\setminus \left\{n^{\prime }\right\}$. So, we can let $p\left(n_j\right)=p^{\prime }\left(n_j\right)$ at all remaining realisations $n_j$. Then (4) reduces to

$V_n^i\left(z^{\prime }\right[p\left]\right)P\left(n^{\prime }|n\right)\ge V_n^i\left(z^{\prime }\right[p^{\prime }\left]\right)P\left(n^{\prime }|n\right)$

$V_n^i\left(z^{\prime }\right[p\left]\right)\ge V_n^i\left(z^{\prime }\right[p^{\prime }\left]\right)$. (The probability is nonzero as $n^{\prime }$ is realised by assumption.)

This holds via (1), so $p\in C^{\text{SM}}\left(\Omega \right(T,n\left)\right)$ and, by Lemma 2, $p\in D\left(\Omega \right(T,n\left)\right)$. $■$

We have thereby shown that a naive DSM agent is strongly dynamically consistent in a very broad class of cases. Although these results are restricted to PIP-uncertainty, this applies to agents with complete preferences too. It’s just a result of the forward-looking nature of naive choice. In the next section, I will drop the PIP restriction by employing a different dynamic choice principle.

2.4.  Sophisticated Choice

Sophistication is the standard principle in dynamic choice theory. It achieves dynamic consistency in standard cases by using backward induction to achieve the best feasible outcome. (This is also the method by which subgame perfect equilibria are found in dynamic games of perfect information.) However, backward induction is undefined whenever the prospects being compared are either incomparable or of equal value. Rabinowicz (1995) proposed a ‘splitting procedure’ to address this, and Asheim (1997) used a similar approach to refine subgame perfection in games.

According to this procedure, whenever there is a tie between multiple prospects, the agent will split the sequential decision problem into parts, where each part assumes that a particular prospect was chosen at that node. The agent then compares each part’s solution and picks its favourite as the solution to the grand decision problem. Intuitively, these agents will follow a plan as long as they lack “a positive reason for deviation” (Rabinowicz 1997). In other words, the agent will consider all the plans that would not make it locally act against their own preferences and, among those plans, proceed to pick the one with the best ex-ante outcome.

In the case of incomplete preferences, however, it turns out that strongly maximal sophisticated choice with splitting will not suffice to guarantee opportunism. The reason behind this is that strong maximality does not satisfy Set Expansion, and that backward induction makes local comparisons.^[11]

Proposition 4   Strongly maximal sophisticated choice with splitting does not guarantee opportunism for Suzumura consistent agents. [Proof.]

DSM, however, will suffice.

Proposition 5 (Strong Dynamic Consistency Under Certainty via Sophistication)
Assume certainty and Suzumura consistency. Then DSM-based backward induction with splitting reaches a strongly maximal terminal node. [Proof.]

More importantly, we can guarantee this property under uncertainty even without PIP.

Proposition 6 (Strong Dynamic Consistency Under Uncertainty via Sophistication)
Let $T$ be a Bayesian decision tree in which $n$ is a non-terminal node, $n^{\prime }\in N_{+}\left(n\right)$ is a successor, and $p\in \Omega (T,n)$ is consistent with $S\left(n^{\prime }\right)$. Assume Material Planning and Preference Stability. Then with DSM-based backward induction (DSM-BI), $p$ is permissible at $n$ iff $p$ is permissible at $n^{\prime }$.

Proof.  We begin by formalising DSM-BI. For tree $T$, let $X$ denote its terminal nodes, $C$ its choice nodes, and $N$ its natural nodes. Let $I[n,A]=\{p\in \Omega (T,n):p(n^{\prime })\in A\}$ denote the set of plans available at $n$ that are consistent with some associated plan continuation in $A$. We then define the tree’s permissible set of plans, $\Psi \left(T\right)$, recursively:

1. $\tilde{\Psi }(T,n)=\left\{S\right(n\left)\right\}$ for $n\in X$.
2. $\tilde{\Psi }(T,n)=D\left(I\right[n,\Phi \left(n\right)\left]\right)$ where $\Phi \left(n\right)={\bigcup }_{n_i\in N_{+}\left(n\right)}\tilde{\Psi }(T,n_i)$ , for $n\in C$.
3. $\tilde{\Psi }(T,n)={\bigcap }_{n_i\in N_{+}\left(n\right)}I[n,\tilde{\Psi }(T,n_i\left)\right]$ for $n\in N$.

The permissible set of plans is then given by $\Psi \left(T\right)\triangleq \tilde{\Psi }(T,n_0)$. Notice that this is the set of plans consistent with backward induction using DSM, and that splitting is implicit since each pruning of the decision tree is set-valued.

Now suppose that $p\in \tilde{\Psi }(T,n)$ for some $n\in C$.

Then $p\in D\left(I\right[n,\Phi \left(n\right)\left]\right)$ , where $\Phi \left(n\right)={\bigcup }_{n_i\in N_{+}\left(n\right)}\tilde{\Psi }(T,n_i)$.

So necessarily $p\in I[n,\Phi (n\left)\right]$. By construction of $I[\cdot ,\Phi (\cdot \left)\right]$, we know that $p\left(n^{\prime }\right)\in \Phi \left(n\right)$.

And since $\Phi \left(n\right)=\tilde{\Psi }(T,n^{\prime })\cup {\bigcup }_{n_i\ne n^{\prime }}\tilde{\Psi }(T,n_i)$, we have $p\left(n^{\prime }\right)\in \tilde{\Psi }(T,n^{\prime })$.

Next, suppose that $p\in \tilde{\Psi }(T,n)$ for some $n\in N$. Then $p\in {\bigcap }_{n_i\in N_{+}\left(n\right)}I[n,\tilde{\Psi }(T,n_i\left)\right]$ . So, for any particular $n^{\prime }\in N_{+}\left(n\right)$, $p\in I[n,\tilde{\Psi }(T,n^{\prime }\left)\right]$. Therefore $p\left(n^{\prime }\right)\in \tilde{\Psi }(T,n^{\prime })$ as needed.

Having established $p\in \tilde{\Psi }(T,n)⟹p\left(n^{\prime }\right)\in \tilde{\Psi }(T,n^{\prime })$, we proceed to the converse. ($\ast$)

Notice that the only nodes reachable via sophisticated choice are those that are consistent with some plan $p\in \Psi \left(T\right)$ which was initially permissible. Therefore, for any $n^{\prime }$ that is reached, there must be some $p\in \Omega (T,n)$, consistent with $n^{\prime }$ where $n^{\prime }\in N_{+}\left(n\right)$, which satisfies $p\in \tilde{\Psi }(T,n)$.

Suppose $p\left(n^{\prime }\right)\in \tilde{\Psi }(T,n^{\prime })$ for some reachable $n^{\prime }\in N$. Whether $n$ is a choice or a natural node, we know by construction of (2) and (3) that

    $\tilde{\Psi }(T,n)\subseteq I[n,{\bigcup }_{n_i\in N_{+}\left(n\right)}\tilde{\Psi }(T,n_i)]$.

Therefore, if $n^{\prime }$ is reached, then the permissible plans at $n^{\prime }$ must be continuations of plans that were permissible at $n$. Hence $p\left(n\right)\in \tilde{\Psi }(T,n)$.

Together with ($\ast$), this establishes that $p\in \tilde{\Psi }(T,n)$ iff $p\left(n^{\prime }\right)\in \tilde{\Psi }(T,n^{\prime })$. $■$

This lets us achieve strong dynamic consistency even in cases where the probability of an event depends on what the agent plans to do after the event has occurred. An example of such a decision problem is Sequential Transparent Newcomb, as devised by Bryan Skyrms and Gerard Rothfus (cf Rothfus 2020b). So, even in this very general setting, our agent’s incomplete preferences aren’t hindering its opportunism.

2.5.  Resolute Choice

A final canonical principle of dynamic choice is resoluteness as introduced by McClennen (1990). An informal version of this principle is often discussed in the AI alignment community under the name ‘updatelessness’. Briefly, a resolute agent will, first, pick its favourite trajectory now under the assumption that this plan will continue to be implemented as the agent moves through the decision tree. And, second, the agent will continually implement that plan, even if this makes it locally choose counter-preferentially at some future node.^[12] It respects its ex-ante commitments.

Under certainty or uncertainty, this is the easiest principle with which to guarantee the invulnerability of agents with incomplete preferences (relative to those with complete preferences). By construction, the agent never implements a plan that was suboptimal from an earlier perspective. I will therefore omit formal derivations of dynamic consistency for resolute choosers.^[13]

3.  The Trammelling Concern

A possible objection to the relevance of the results above is what I’ll call the Trammelling Concern. According to this objection, agents with incomplete preferences who adopt DSM as a dynamic choice rule will, in certain sequential choice problems, or over the course of sufficiently many and varied ones, eventually converge in behaviour to an agent with complete preferences using Optimality as a choice rule. This would be worrying since complete preferences and optimal choice resembles the kind of consequentialism that precludes Thornley-style corrigibility.

This section aims to quell such worries. I begin with a taxonomy of our agent’s possible doxastic attitudes towards decision trees. First, the set of non-terminal nodes of a given tree will either contain only choice nodes, or it will contain some natural nodes. Second, the structure of a given tree will either be entirely known to the agent, or it will not. If it is not known, the agent will either have beliefs about all its possible structures, or its credence function will not be defined over some possible tree structures. We will refer to these three cases as certainty, uncertainty, and unawareness about structure, respectively.^[14] Finally, at least under unawareness, the possible tree structures may or may not include terminal nodes (prospects) that are also present in the tree structure currently being considered by the agent. I will say the prospects are new if so and fixed if not.

This table summarises the situations our agent might find itself in and its cells indicate whether trammelling can occur in each case. This is based on the arguments and results below. The conclusions for (1)-(2) are based on conceptual considerations in section 3.1. Cases (3)-(4) are not discussed explicitly since uncertain tree structures can simply be represented as certain tree structures with natural nodes. The conclusions for (5) and (7) are based on formal results in section 3.2. For the sake of brevity, the effects of natural nodes under unawareness are left for later work.

3.1.  Aware Choice

This section was co-authored with Eric Chen.

Let’s begin with an important distinction. As Bradley (2015) puts it, there’s a difference between "the choices that are permissible given the agent’s preferences, those that are mandatory and those that she actually makes" (p. 1). If an agent is indifferent between two options, for example, then it can be the case that (i) both are permissible, (ii) neither is mandatory, and (iii) a particular one is in fact chosen. One aspect of choice between incomparables that we need to preserve is that all were permissible ex-ante. The fact that one is in fact chosen, ex-post, is immaterial.

To see what this implies, consider first the case of certainty. Here we proved that DSM will result in only ex-ante strongly maximal plans being implemented (Proposition 2; Corollary 1; Proposition 5). Now consider the following toy case (Example 1). A Suzumura consistent agent can’t compare $A$ and $B$. It’s offered a choice between the two. Suppose it chooses $B$. Does this constrain the agent to never pick $A$ in future choices between $A$ and $B$?

No. In fact, it’s a bit unclear what this would mean formally. Once we expand the decision tree to include future choices between $A$ and $B$ in some branches, everything boils down to the fact that all plans through this tree will result in either $A$ or $B$. And so any plan will be permissible. What DSM constrains is which terminal nodes are reached; not how the agent gets there. Let’s see how far this reasoning can get us.

Example 2   A Suzumura consistent agent’s preferences satisfy $A\succ A^{-}$ and $B\bowtie /A,A^{-}$. It’s offered a choice between going ‘down’ for $A$ or going ‘up’ to face a choice between $A^{-}$ and $B$. It happens to go ‘up’. Does this constrain the agent not to pick $A^{-}$ over $B$?

This is a case of an agent ending up on some trajectory that is only consistent with one of its coherent extensions ('completions'). One might worry that, at this point, we can treat the agent as its unique extension, and that this denies us any kind of corrigible behaviour. But there is a subtle error here. It is indeed the case that, once the agent has reached a node consistent with only one extension, we can predict how it will act going forward via its uniquely completed preferences. But this is not worrisome. To see why, let’s look at two possibilities: one where the agent is still moving through the tree, and another where it has arrived at a terminal node.

First, if it is still progressing through the tree, then it is merely moving towards the node it picked at the start, by a process we are happy with. In Example 2, option $A^{-}$ was never strongly maximal in comparison to $A$ and $B$. We knew, even before the agent went ‘up’, that it wouldn’t pick $A^{-}$. So the agent is no more trammelled once it has gone ‘up’ than it was at the beginning.

For a more concrete case, picture a hungry and thirsty mule. For whatever reason, this particular mule is unable to compare a bale of hay to a bucket of water. Each one is then placed five metres from the mule in different directions. Now, even if the mule picks arbitrarily, we will still be able to predict which option it will end up at once we see the direction in which the mule is walking. But this is clearly not problematic. We wanted the mule to pick its plan arbitrarily, and it did.

Second, if the agent has reached the end of the tree, can we predict how it would act when presented with a new tree? No; further decisions just constitute a continuation of the tree. And this is just the case we described above. If it did not expect the continuation, then we are simply dealing with a Bayesian tree with natural nodes. In the case of PIP-uncertainty, we saw that DSM will once again only result in the ex-ante strongly maximal plans being implemented (Proposition 3).^[15] Here it's useful to recall that plans are formalised using conditionals:

    $p\left(n\right)=S\left(n\right)\wedge {\bigwedge }_{j=1}^{|N_{+}\left(n\right)|}\left[S\right(n_j)\to S(z_j\left)\right]$, where $n_j\in N_{+}\left(n\right)$ and $z_j\in N_{+}\left(n_j\right)$.

That is, plans specify what path to take at every contingency. The agent selects plans, not merely prospects, so no trammelling can occur that was not already baked into the initial choice of plans. The general observation is that, if many different plans are acceptable at the outset, DSM will permit the agent to follow through on any one of these; the ex-post behaviour will “look like” it strictly prefers one option, but this gets close to conflating what options are mandatory versus actually chosen. Ex-post behavioural trajectories do not uniquely determine the ex-ante permissibility of plans.

3.2.  Unaware Choice

The conceptual argument above does not apply straightforwardly to the case of unawareness. The agent may not even have considered the possibility of a certain prospect that it ends up facing, so we cannot simply appeal to ex-ante permissibility. This section provides some results on trammelling in this context. But, first, given the argument about Bayesian decision trees under certain structure, one may ask: what, substantively, is the difference between the realisation of a natural node and awareness growth (if all else is equal)? Would the behaviour of our agent just coincide? I don't think so.

Example 3   Consider two cases. Again, preferences satisfy $A\succ A^{-}$ and $B\bowtie /A,A^{-}$.

Up top, we have a Bayesian tree. On the bottom, we have awareness growth, where each tree represents a different awareness context. In both, let’s suppose our agent went ‘up’ at node 0. In the Bayesian case, let’s also suppose the natural node (1) resolves such that the agent reaches node 2. In both diagrams, the agent ends up having to pick $A^{-}$ or $B$.

The permissible plans at the outset of the Bayesian tree are those that lead to $A$ or to $B$ (since $A^{-}$ is strictly dominated by $A$). But in the bottom case, it seems that our agent faces two distinct decision problems: one between $A$ and $B$; one between $A^{-}$ and $B$. In the first problem, both options are permissible. And likewise in the second. So, the difference between these situations is that, in the latter case, our agent faces a new decision problem.

In the Bayesian case, our agent might end up at a future node in which the only available options are incomparable but nevertheless reliably picks one. But the reason for this is that another node, also available at the moment the agent picked its plan, was strictly preferred to one of these. The agent’s decision problem was between $A$, $A^{-}$, and $B$. The arbitrary choice was between $A$ and $B$. This is as it should be; there is no trammelling.

However, in the case of unawareness, the agent initially chose between $A$ and $B$ and now it must choose between $A^{-}$ and $B$. It never got to compare $A$ to $A^{-}$ to $B$. It compared $A$ to $B$ and, separately, $A^{-}$ to $B$. Therefore, if it reliably picks $B$ over $A^{-}$, for the sake of opportunism, one could see this as a form of trammelling. I now turn to this issue.

For brevity, I will focus on cases 4 and 5 from the taxonomy above. In these worlds, all decision trees have choice nodes and terminal nodes, but they lack natural nodes. And our agent is unaware of some possible continuations of the tree they are facing. That is, it’s unaware of—places no credence on—the possibility that a terminal node is actually another choice node. But once it in fact gets to such a node, it will realise that it wasn’t terminal. There are then two ways this could play out: either the set of available prospects expands when the tree grows, or it doesn’t. Let’s consider these in turn. As we’ll see, the most salient case is under a certain form of expansion (section 3.2.3., Example 6).

3.2.1.  Fixed Prospects (No Trammelling)

Here I’ll show that when prospects are fixed, opportunism and non-trammelling are preserved. We begin with an illustrative example.

Example 4   The agent’s preferences still satisfy $A\succ A^{-}$ and $B\bowtie /A,A^{-}$.

The agent initially sees the top tree. DSM lets it pick $A$ or $B$, arbitrarily. Suppose it in fact goes for $A$. Then its awareness grows and the tree now looks like the bottom one. It can stay put (go ‘down’ to $A$) or move ‘up’ and choose between $B$ and $A^{-}$. DSM still says to pick, arbitrarily, any path leading to $A$ or to $B$. Suppose it goes for $B$. There—no trammelling.

Here’s the lesson. Our agent will embark on a path that’ll lead to a node that is DSM with respect to the initial tree. When its awareness grows, that option will remain available in the new tree. By Set Contraction (Lemma 4) this option will also remain DSM in the new tree. So, picking it will still satisfy opportunism. And regarding trammelling, it can still pick entirely arbitrarily between all initially-DSM nodes. DSM constrains which prospects are reached; not how to get there. We can state this as follows.

Proposition 7   Assume no natural nodes. Then, DSM-based choice under awareness growth will remain arbitrary whenever the set of available prospects is fixed. [Proof.]

3.2.2.  New Prospects (No Trammelling)

Now consider the case where the set of available prospects expands with the tree. It's of course possible that our agent gets to a terminal node that isn’t strongly maximal according to the new tree. This could happen if they get to an initially-acceptable terminal node and then realise that a now-inaccessible branch would let them access a prospect that’s strictly preferred to all those in the initial tree.^[16] But, importantly, this applies to agents with complete preferences too. It’s simply an unfortunate feature of the environment and the agent’s epistemic state. Complete preferences don’t help here. 

But there is a class of cases in which completeness, at first glance, seems to help.

Example 5   The agent’s preferences satisfy $A\succ A^{-}\sim B$ and $A\bowtie /B$ (unlike before).

The agent sees the tree on the left. DSM lets it pick $A$ or $B$, arbitrarily. Suppose it in fact goes for $B$. Now its awareness grows and the tree looks like the one on the right. It can stay put (go ‘down’ to $B$) or move ‘up’ to get $A^{-}$.

Now, none of the available options are DSM in the full tree, but the agent still has to pick one. This, on its own, is not problematic since it could just be an unfortunate aspect of the situation (as dismissed above). But in this particular case, because an agent with complete preferences would satisfy transitivity, it would never go for $B$ in the first place. This kind of issue wouldn’t occur if our agent’s incomplete preferences were transitive, rather than just strongly acyclic.

So let’s impose transitivity. Then our agent wouldn’t pick $B$ in the first place. In general it will now remain opportunistic in the same way that its complete counterparts are. That’s because it would never initially choose arbitrarily between options that all its coherent extensions have a strict preference between. So it would never pick in a way that makes it realise that it made a foreseeable mistake once more options come into view. To operationalise this, I'll say that an agent facing deterministic decision trees fails to be opportunistic under awareness growth iff

1. The agent fails to reach a strongly maximal prospect in the grand tree, and
2. A strongly maximal prospect in the grand tree was available in the initial tree.

This agent is opportunistic otherwise.^[17] We can now state the associated result.

Proposition 8  Assume transitive preferences and no natural nodes. Then, under all possible forms of awareness growth, naive choice via DSM is opportunistic. [Proof.]

That gives us opportunism, but what about non-trammelling? Will our agent ever have to pick one option over another despite them being incomparable? Yes, in some cases.

3.2.2.  New Prospects (Bounded Trammelling)

Example 6   The agent’s preferences satisfy $A\succ A^{-}$ and $B\bowtie /A,A^{-}$.

It faces the same trees as in the previous example. DSM first lets it pick $A$ or $B$, arbitrarily. Suppose it goes for $B$. Now its awareness grows and the tree looks like the one on the right. It can stay put (go ‘down’ to $B$) or move ‘up’ to get $A^{-}$. Since only $B$ is DSM in the global tree, our agent might reliably pick $B$ over $A^{-}$, despite them being incomparable. Trammelled.

This is a special property of unawareness. Thankfully, however, we can bound the degree of trammelling. And I claim that this can be done in a rather satisfying way. To do this, let’s formally define an extension of DSM that will give us opportunism under awareness growth. Let $T_1$ denote the initial tree, with initial node $n_0$, and $T_2$ the tree available after awareness growth, with initial node $n^0$. And let $T$ denote the appending of $T_2$ on $T_1$.